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ABSTRACT 

Background: The resources, needs and implementation activities of educational projects are often 
straightforward to document, especially if objectives are clear. However, developing appropriate 
metrics and indicators of outcomes and performance is not only challenging but is often overlooked 
in the excitement of project design and implementation. The authors will show how this problem can 
been addressed using the Irish Integrative Learning Project (IILP) as an example. The goals of this 
NAIRTL-funded project are to help students become integrative thinkers and learners. Educational 
capacity is being addressed through fourteen multi-institutional and multi-disciplinary teaching 
initiatives to act as stimuli for furthering Integrative Learning in Ireland. 

Aims: The purpose of this paper is to demonstrate how Outcomes Logic Model (OLM) can help develop 
clarity of thinking and targets in educational projects. 
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Method: OLM is a systematic visual way to present a pLanned programme with its underlying 
assumptions and theoretical framework. OLM allows us to describe, share, discuss and 
improve programme theory, in words and pictures. It can be used at any stage, during design, 
planning, implementation, evaluation and reporting. It can strengthen the case for programme 
investment (grants). It can also reflect group processes and change over time. 

Results: We will present our completed OLM for the IILP using the OLM framework. While 
outputs are relatively easy to develop, measurable short and long-term outcomes pose 
significant challenges. 

INTRODUCTION 

"Government tikes to begin things — to declare grand new programs and causes. 

But good beginnings are not the measure of success. What matters in the end is 
completion. Performance. Results." (The Kellogg Foundation, 2001). 

In this era of 'scientific education' research, funding bodies and employers are demanding 
evermore transparency and accountability, in addition to evidence of effectiveness, efficiency 
and good educational practice. For these reasons, educators need a structured, systematic 
approach to developing, implementing and evaluating educational research projects. 


The Irish Integrative Learning Project (IILP) was developed as a NAIRTL-funded, multi- 
institutional, multi-disciplinary research project to promote small-scale research that sought 
to foster students' integrative learning in higher education institutions in Ireland. The 
project planned to develop and sustain a learning community of teachers who investigate 
and document examples of students' integrative learning, and who produce clear and 
practical integrative learning resources for all teachers. The project initially involved 
eighteen academics (sixteen Irish members and two international Associate Members), three 
collaborating institutions (University College Cork, Waterford Institute of Technology and the 
Law Society of Ireland) and eleven disciplines (Art History, Arts and Drama, Applied Maths, 
Economics, Geology, Law, Occupational Therapy, Paediatrics, General Practice, Nursing and 
Zoology (Appendix 1, Table 2). 

The aim of this paper is to introduce the Outcomes Logic Model concept and describe how it 
was used to envisage, design, develop, implement and evaluate the Irish Integrative Learning 
Project. The objectives were to provide guidance and support for the project participants as 
well as coordinate dissemination of their research findings. 

THE OUTCOMES LOGIC MODEL (OLM) 

The Outcomes Logic Model was developed by the W.K. Kellogg Foundation (2001) to serve as 
a platform for all its research applications. It is a framework for organising thoughts and for 
guiding the researcher through the structure and purpose of the project and documenting to 
what extent important outcomes are achieved. The OLM also helps stakeholders to know what 
the project is intended to deliver and what impact it is intended to create (Alter and Murty, 
1997; Conrad and Randolph, 1999; Hernandez, 2000; Julian, 1997; McLaughlin and Jordan, 
1999; Stinchcomb, 2001; Unrau, 2001). The importance of having a priori measurable research 
outcomes in any educational project (i.e. 'designing the project backward') is emphasised in 
this approach. The OLM that was used from the outset of the Irish Integrative Learning Project 
is presented in Appendix 1 (Table 1). The template consists of five columns, derived from three 
broad themes: Antecedents (resources, context and stakeholders of the project); Process 
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(activities needed to implement the project) and Evaluation (outputs, outcomes and impacts of the 
project). 

This OLM approach encouraged us to clarify the objectives of the project, and articulate what exactly 
we were trying to achieve and how we would measure the impact. We used collaborative workshops 
to build a shared meaning of integrative learning as a theoretical concept, to list the key attributes 
of an integrative learner, and to explore the implications for curriculum design. In addition, we 
re-examined the objectives as set out in the initial project proposal, discussed and collated the 
disciplinary research initiatives, and re-identified meaningful and measureable outcomes with realistic 
timelines. 


We noted that promoting integrative learning involves an approach to curriculum design and 
pedagogy that is intended to help learners make connections between their sometimes fragmentary 
learning experiences. Integrative learning " comes in many varieties: connecting skills and knowledge 
from multiple sources and experiences; applying theory to practice in various settings; utilizing diverse 
and even contradictory points of view; and , understanding issues and positions contextually" (Huber 
and Hutchings, 2004, p. 13). As our workshop series progressed, we realised we were beginning 
to become more integrative in our own thinking, and more intentional in our curriculum planning 
for integrative learning. These discussions helped develop a language for integrative learning, 
and opened up more questions for the group such as: How will I assess integrative learning in my 
discipline? 


We agreed that with modularisation and increased mobility students can have fragmentary learning 
experiences in their third level education. As a result, they may fail to make meaningful connections 
within and between subjects and disciplines. If knowledge becomes 'troublesome' students may have 
difficulty grasping the key disciplinary concepts - threshold concepts - that are essential for their 
development (Meyer and Land, 2003). When learning becomes integrative, threshold concepts can 
be negotiated by students, allowing them to advance in the construction and application of their 
knowledge. The whole Irish Integrative Learning Project was underpinned by a concern expressed by 
Klein (2005, p. 10): " The answers students seek and the problems they will need to solve as workers, 
parents and citizens are 'not in the book'. We wished to build students' capacities to connect-up 
and integrate their learning by providing opportunities that encouraged all students to carry their 
disciplinary skills from one learning landscape to another. The recurring question was: How can we 
help students to think and link - make connections and become integrative learners? 


Having identified some of the challenges of the integrative learning concept, the OLM encouraged 
us to identify our target audience, potential stakeholders and our assets (Appendix 1, Table 1). We 
began to realise that our audience was in many ways ourselves, the educators. We were largely novice 
but self-motivated learners of the concept of integrative learning. Potential stakeholders included 
our institutions, disciplines, departments and colleagues, but our students were deemed our most 
important stakeholders. We explored questions they might ask and assumptions we might be making 
about them. 

Our most important assets were the broad and diverse experiences of the project participants, and 
the prospect of multi-institutional and interdisciplinary interactions. It was decided that the work of 
these participants should include consideration of curriculum-design, pedagogy, assessment, public 
policy and community involvement. Recognizing the limited protected time available to participants 
for educational research, we expected the disciplinary initiatives would address questions that could 
be explored as part of the everyday work, and within the classrooms of the various participants. Thus, 
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our projects were subject-centred and authentic as advocated by Kreber (2007). 

Thinking strategically and systematically, through the OLM approach, we identified a number 
of influential factors that could determine the success of our project. We noted the resurgent 
emphasis on the scholarship of teaching and learning in our institutions. This was an 
influential driver for the project. A competitive NAIRTL grant award (€20,000) was essential in 
driving the project. At participant level there was a desire to improve the students' experience 
by helping them overcome the potential fragmentation as a consequence of modularisation and 
mobility. 


In implementing the Irish Integrative Learning Project, we referred to good practices used 
by other groups (Carnegie Academy for the Scholarship of Teaching and Learning, and the 
Centres for Integrative Learning in the University of Nottingham and Active Learning in the 
University of Gloucester in the UK). The project leaders' main functions during the twelve 
month implementation period, were to maintain the management plan, revise time-lines, and 
sustain connections with and between the participants through structured meetings, e-mail, 
web-postings and dialogue on the NAIRTL/IILP website. As an essential driver, the concept 
of Critical Friends was introduced at an early stage. A critical friend is a trusted person who 
asks provocative questions, allows research data to be examined through another lens, offers 
critique of a person's work and is an advocate for the success of that work. (Costa and Kallick, 
1993). Participants were matched as potential critical friends, according to practicality, 
compatibility and availability. Reports and reflections on the contacts between critical friends 
were posted to a project workspace on the NAIRTL website. Prof. Alan Booth, an international 
associate, was invited to critique and support the individual project participants through one- 
to-one meetings. 


It was essential, during each project workshop, to familiarise ourselves with the distinctions 
between outputs, outcomes and impacts as conceived by the OLM. An Output is the number of 
'what was created and what was delivered' by the activities of the project. We had no problems 
in documenting outputs (Apendix 1, Table 1). Relevant outputs include NAIRTL reports and 
documentation of the disciplinary research initiatives in the form of book chapters, journal 
articles, and a planned symposium to disseminate the findings and expand the integrative 
learning network. Newly designed course assessments and rubrics, developed by the project 
participants are outputs and are being made publicly available. 

An Outcome refers to a behavioural change in people (knowledge, attitudes or skills) of 
an organisation, in this case engendered by the Irish Integrative Learning Project. Staff 
development, and the impact that has on student learning, were the most important short- 
term outcomes of this project. The participants have connected with other academics 
institutionally, nationally and internationally. The research projects, the new knowledge 
and understandings generated, and the motivation to complete and continue, showed that 
participants were transformed in small but significant ways by their involvement in the Irish 
Integrative Learning Project (Appendix 1, Table 3). In their writings, researchers showed a 
deepening understanding of the nature of integrative learning, and how it can be promoted. 
They developed a new language with which they can debate, consolidate and disseminate their 
teaching practice. They have become more intentional in their teaching, and are documenting 
the elements and activities that nurture students' learning. Pedagogies and teaching strategies 
known to provide rich opportunities for integrative learning, including problem-based learning, 
reflective-portfolios and critical friends, are being used (Appendix 1, Table 1). They pursued 
scholarly approaches to collecting and analysing evidence for 'opportunities to connect' in 
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their research projects. They have shared these insights and ideas as well as strategies to clarify their 
pedagogical goals and how connections can be strengthened at multiple levels. They have observed 
important changes in student behaviour as a result of their integrative learning activities (Appendix 
1, Table 3). 

The most important and the most difficult outcomes to measure for any project are its impacts or 
seven to ten year outcomes. Our most important outcome was the bringing together of a diverse, 
multi-institutional, multi-disciplinary group of participants: in other words the development of 
a learning community of integrative learning teachers and educational researchers. We expect 
that there will be increasing inter- and intra-faculty discussions that will advance institutional 
understanding and the value of this integrative learning community. We believe that the 
understandings gained by teaching staff in this project will continue to enhance student autonomy, 
allowing them to continue to make valuable connections throughout their lives. 


In its essence, the OLM is a tool to promote better thinking and to plan with the end in mind. It has 
kept the Irish Integrative Learning Project focused and on track. It has reminded the participants 
to 'Clarify Your Outcomes First', and to strive to maintain alignments between the project's aims, 
process and outcomes. Finally, as recommended by Huber and Hutchings (2004), the OLM enabled us 
to catalyse and encourage teachers into the 'big tent' of the Scholarship of Teaching and Learning, by 
valuing small efforts based on reflection of one's own teaching and sharing what is learned. 
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APPENDIX 1 


Table 1: Outcomes Logic Model for the Irish Integ 

rative Learning Project 

1 . 

Resources 

2. 

Activities 

3. 

Outputs 

4. 

Outcomes 

Short-Medium 

Term 

5. 

Impact 

Long-Term 

Outcomes 

Definitions of 

integrative 

learning 

Shared 
Meaning of 
integrative 
learning 

Context of 
IILP 

Content of 

IILP 

Why 

integrative 
learning now? 
The specific 
aims and 
objectives? 
The 

Stakeholders? 

Assumptions 

about 

Participants & 
Students? 
NAIRTL Grant 

Preparatory 

work 

5 one day 

workshops 

Invited 

speakers 

Discussion in 

workspace on 

Web-page 

E-Learning 

Critical 

Friends 

PLanning 

symposium 

Writing 

chapters for 

book 

Writing 

reports 

Project 

evaluation 

Number of 

Participants 

(18) 

One-day 
workshops (5) 
Disciplinary 
initiatives (14) 

Students 
affected 1000 
(estimate) 
National/ 
international 
presentations 
(6) 

NAILTL reports 
(2) 

Book Chapters 
(12) providing a 
resource for saff 
Published 
papers (3) 

Hours of 
teaching 
New Learning 
& Assessment 
Materials 
Founding of 
Integrative 
Learning 
Community 

14 research 
initiatives 
showing evidence 
of: 

Changes in 
pedagogical style 
& content 
New knowledge 
& understandings 
created (teachers 
and students) 

• Student 
centered 
teaching 

• Student 
engagement 

• Integrative 
learning capacity 
building 

National & 
International 
Multi- 
disciplinary 
IL Learning 
Community 
Discipline- 
based 

programmes 
with IL as an 
explicit goal 

Students 
carrying 
and using 
integrative 
approaches 
beyond 
their formal 
training 
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Table 2: List of Participants, Disciplines and Integrative Learning Projects 


Participant 

Project Title 

Daniel Blackshields 
Economics, UCC 

Student Reflective Journals: Scaffolding an 
Autobiographical Approach to Economics 
Education 

Maura Butler 

Law Society of Ireland 

The Confluence of Professional Legal 
Training, ICT and Language Learning Towards 
the Construction of Integrative Teaching and 
Learning 

John Considine 
Economics, UCC 

Integrative Learning Through Student 
Behaviour on Assessment 

James Cronin 
History of Art, UCC 

Investigating patterns of new literacy: The 
assumptions we make about university 
entrants into a discipline. 

Bettie Higgs 
Geology, UCC 

Using Threshold Concepts to Build Capacity 
for Integrative Learning in First Year Science 
(UCC) 

Martina Kelly 
Medical School UCC 

Case records as a means of integrated 
assessment in Medical Students 

Tom Kelly & Michael O'Callaghan 
Zoology, Ecology, and Plant Science, and 
Applied Math, UCC 

The application of applied Mathematics to 
Biology: an Integrative Learning Project 

Shane Kilcommins 
Law UCC 

The Use of Learning Journals in Legal 
Education 

Marian McCarthy 
Education, UCC 

The Arts in Education as an Integrative 
Learning Approach 

Pat Meere 
Geology UCC 

Integrative Learning in Geology 

F. Catherine Pettigrew 
Speech & Language, UCC 

Facilitating learning by integrating 
knowledge and skills from different sources: 
Speech and Language Therapy students' 
perceptions 

Walter O'Leary & Sinead Cenneelly 
Law, WIT 

Problem Based Learning on a new degree 
programme 

Tony Ryan 
Medical School, UCC 

Drawing on Medical Students drawings 
to illuminate concepts of Humanism and 
Professionalism 

Nuala Walsh 
School of Nursing, UCC 

Integrative Learning in Nursing Studies 


Table 3: Impact of integrative learning experiences on group participants, captured during the 
fourth Integrative Learning Project workshop. 


Teachers no longer 
afraid of loss of 
control 

You have no idea of where the students are going to go. 
You realise the importance of letting go of control. 

I no longer am afraid when I don't know everything. 

It excites me when students know something I don't know. 

Guidance of students 

You realise the importance of getting students into the mindset - 
scaffolded, prompted, guided by us. 

I have become more facilitative- more aware of my function as a role 
model. 

Self-development 
as an Integrative 
Teacher 

Now I am more strategic in my preparation. 

I ask "What do I want them to know, and why?" 

Achieving higher order thinking is rewarding to the teacher as well 
as student. 

I have become more metacognitive in terms of my teaching. 

As a teacher. I'm not done with learning yet! 

If we want our students to change, we, as educators, must also 
change. 

We can help students to make connections & integration through an 
intentional, democratic approach to teaching. 

We need to recognize the importance of context. 

We need to change our assessment methodologies. 

We must try to identify discipline specific Threshold Concepts. 

Teacher observations 
on students, 
following 

integrative learning 
experiences 

Student attendance has increased; their confidence and interest has 
increased. 

Students who were not turning up - are now turning up. 

I was observing the students more, and how they reacted. 

We tried it [integrating Maths and Zoology]. We saw an excitement, 
a buzz. 

There is a Lot of overlap between what we got out of it and what the 
students got out of it. 

I would never have done that before [I now explain what I want 
them to do with their reflective journals]. 

Student said "It was my first time ever that I had used economics" 
When you have used it, it is yours forever. 

Students can be confused by integrative Learning if there is a 
misalignment between theory and practice, if there is a disconnect. 
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